Conversation
More templates
@tanstack/angular-db
@tanstack/db
@tanstack/db-browser-wa-sqlite-persisted-collection
@tanstack/db-ivm
@tanstack/db-react-native-sqlite-persisted-collection
@tanstack/db-sqlite-persisted-collection-core
@tanstack/electric-db-collection
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
|
Size Change: +504 B (+0.46%) Total Size: 111 kB
ℹ️ View Unchanged
|
|
Size Change: 0 B Total Size: 4.23 kB ℹ️ View Unchanged
|
Document the transactional metadata model for persisted collections, including row and collection metadata, query retention, and Electric resume state. Made-with: Cursor
Break the persisted sync metadata RFC into phased implementation docs covering the core API, SQLite integration, query collection, Electric collection, and required invariants tests. Made-with: Cursor
Tighten the RFC and phased plan around startup metadata reads, query-owned reconciliation, cold-row retention cleanup, replay fallback behavior, and Electric reset semantics. Made-with: Cursor
Add transactional row and collection metadata plumbing across core sync state, SQLite persistence, query collections, and Electric resume state so persisted ownership and resume metadata survive restarts. Made-with: Cursor
Buffer persisted metadata writes within wrapper transactions and dedupe concurrent collection setup so warm starts no longer trip missing sync transaction errors or collection registry races. Made-with: Cursor
fce939f to
7591ee1
Compare
Drop the RFC and phased implementation plan from the branch while leaving the local working copies in place. Made-with: Cursor
Finish the core persisted metadata follow-through so reloads, retained query ownership, and Electric resume/reset state behave correctly across startup and recovery while clarifying metadata semantics around inserts and cleanup. Made-with: Cursor
Finish the remaining persisted metadata work by adding cold-row retained query cleanup, runtime TTL expiry, stronger Electric resume identity checks, and metadata delta replay for follower recovery while keeping reload fallback for reset-like cases. Made-with: Cursor
Use the persisted JSON encoder for replay payloads so bigint and date values survive applied_tx serialization and package-level SQLite adapter tests pass under the CLI runtime. Made-with: Cursor
There was a problem hiding this comment.
Correctness: Does It Fix the Bug?
Yes — the core fix is sound. The key change in query.ts is that applySuccessfulResult now computes previouslyOwnedRows from persisted ownership metadata rather than iterating collection._state.syncedData.entries():
const previouslyOwnedRows = shouldUsePersistedBaseline
? new Set(persistedBaseline.keys())
: getHydratedOwnedRowsForQueryBaseline(hashedQueryKey)This means an empty live query only removes rows it previously owned — not rows belonging to history. The ownership metadata (queryCollection.owners) is persisted alongside rows so it survives restarts.
Issues Found
MEDIUM Severity
1. markReady() error handling removed (persisted.ts)
The old code had a .catch() on runtime.ensureStarted() that called params.markReady() on failure:
// Old:
void runtime.ensureStarted()
.then(() => { params.markReady() })
.catch((error) => {
console.warn(`Failed persisted sync startup before markReady:`, error)
params.markReady()
})
// New:
void (fullStartPromise ?? runtime.ensureStarted()).then(() => {
params.markReady()
})If ensureStarted() rejects (e.g., SQLite driver error during hydration or index bootstrap), markReady() is never called, leaving the collection stuck in a permanent loading state. The old code correctly degraded gracefully by marking ready anyway.
2. truncate() preserves collectionMetadataWrites but clears rowMetadataWrites — undocumented asymmetry
In both the core sync.ts and the persisted wrapper createWrappedSyncConfig, the truncate handler clears row-level state but preserves collection-level metadata writes:
openTransaction.operations = []
openTransaction.rowMetadataWrites.clear()
openTransaction.truncate = true
// collectionMetadataWrites NOT clearedThis is intentional — the Electric resume metadata writes happen before truncate and must survive — and tests verify it explicitly. But the asymmetry is subtle and load-bearing. A comment explaining why collection metadata survives truncate would prevent future contributors from "fixing" this.
LOW Severity
3. ALTER TABLE ... ADD COLUMN migration wrapped in bare try {} catch {}
try {
await this.driver.exec(`ALTER TABLE applied_tx ADD COLUMN replay_json TEXT`)
} catch {}This is a standard SQLite migration pattern (SQLite lacks ALTER TABLE ADD COLUMN IF NOT EXISTS), but the empty catch swallows all errors — not just "column already exists." If the driver has a permissions issue or the table is corrupt, the error is lost and the failure would surface later as a confusing "column not found" error.
4. getStableShapeIdentity uses JSON.stringify for identity comparison
function getStableShapeIdentity(shapeOptions) {
return JSON.stringify({ url: shapeOptions.url, params: shapeOptions.params ?? null })
}JSON.stringify iterates own enumerable properties in insertion order, so { table: 'x', where: 'y' } and { where: 'y', table: 'x' } produce different strings. If a user's params object is constructed with different key ordering between sessions, the persisted resume identity would mismatch and trigger a needless fresh sync. The codebase already has stableSerialize in the persisted package that handles this — consider using it here.
5. cancelledLoads uses WeakSet<object> with reference identity
const cancelledLoads = new WeakSet<object>()
// In loadSubset:
cancelledLoads.delete(options as object)
// In unloadSubset:
cancelledLoads.add(options as object)This only works if the exact same options object reference is passed to both loadSubset and unloadSubset. The collection's internal subscription management likely does pass the same reference, but the correctness depends on an undocumented invariant of the caller.
Test Coverage Assessment
Gaps:
- No test for
markReadywhenensureStarted()rejects - No integration test that directly reproduces the original warm-start bug scenario (two disjoint queries with timing-dependent ordering)
Summary
@tanstack/db,db-sqlite-persisted-collection-core,query-db-collection, andelectric-db-collectionHow To Use
Custom sync implementations
If a collection is running on a persistence layer that supports this feature, the sync function now receives an optional
metadataAPI alongsidebegin,write,commit,markReady, andtruncate.Available operations:
metadata.row.get(key)metadata.row.set(key, value)metadata.row.delete(key)metadata.collection.get(key)metadata.collection.set(key, value)metadata.collection.delete(key)metadata.collection.list(prefix?)Behavior:
get/listare allowed before any transaction is openedbegin()/commit()write({ metadata })andmetadata.row.set()target the same row metadata storeSQLite persisted collections
persistedCollectionOptions(...)now persists both row metadata and collection metadata when used withdb-sqlite-persisted-collection-core.Typical usage looks like:
What you get automatically:
Query collections
Query collections can now retain persisted ownership metadata across restarts.
What this enables:
gcTimepersistedGcTimecan be finite or effectively indefinite for offline-first flowsElectric collections
Electric collections now persist resume metadata at collection scope when wrapped with persisted collection options.
What this enables:
offset/handleresume pointTest plan
pnpm vitest --run packages/db/tests/collection.test.tspnpm --filter @tanstack/query-db-collection exec vitest run tests/query.test.ts --coverage.enabled falsepnpm --filter @tanstack/electric-db-collection exec vitest run tests/electric.test.ts --coverage.enabled falsepnpm --filter @tanstack/db-sqlite-persisted-collection-core exec vitest run tests/persisted.test.ts tests/sqlite-core-adapter.test.ts --coverage.enabled falseNotes
cleanupQueryIfIdlewarnings.